A system and method for producing synthetic single or plural word messages was developed by Bruce Baker et al. and is disclosed in U.S. Pat. No. 4,661,916 to Baker et al. (the Baker '916 patent) issued on Apr. 28, 1987, the entire contents of which are hereby incorporated herein by reference. The system was directed to a linguistic coding system and keyboard for use by people with cognitive and/or physical impairments. The coding system and associated keyboard was used to store and access messages, which included words, plural word messages, phonemes, sentences, phrases, full names, letters, numbers, functions, or any combination thereof.
In such a system, the keyboard was coupled to a computer device, or was alternately part of the stand-alone entity which included a microprocessor, memory and display. The memory stored the messages for selective retrieval by the keyboard. The messages retrieved from the keyboard were then fed to a voice synthesizer, for example, which converted them through a loudspeaker to produce audible spoken messages. On this keyboard, associated with each of a plurality of keys, were polysemous (multi-meaning) symbols, also known as icons. By designating selected ones of the keys and their associated symbols or icons, selected stored messages or plural word messages (including but not limited to words, phrases and sentences) were accessed from the memory and then subsequently output.
With the system described in the Baker '916 patent, messages stored in the memory could be retrieved by activating a combination of symbol keys and other keys to vary the context of the polysemous symbols. Thus, a plurality of sentences could be selectively generated as a function of polysemous symbols in combination with other polysemous symbols. This allowed a user the ability to access thousands of words or messages based upon as little as one, two, or three keystrokes. Further, with symbols being polysemous, thousands of symbol sequences could be generated with only a small number of keys on a keyboard. Based upon ease of use of the system, the polysemous icons or symbols utilized, and the easily memorized symbol sequence combinations, such a system became ideal for many mentally and physically challenged users for whom spelling and typing, as well as speech itself, was extremely difficult.
The system of the Baker '916 patent allowed for an operator to go directly from thought to speech. This was possible because each key of the keyboard bore a central image or symbol which was polysemous and illustrated an important aspect of life and/or linguistic function. The keyboards could be varied depending on the intellectual level of the intended operator. Therefore, each keyboard could in itself be a language which was designed for or with a specific user.
Each of the polysemous symbols was developed to be rich in associations so that combinations of symbols could signal sentence or message ideas in the operator's memory. This enabled the generation of plural word or whole sentence messages by the activation of only a limited number of keys. The device allowed for the generation of many sentences or phrases and a large core vocabulary which could be easily retrieved from memory because of the ease with which the polysemous symbols on the keys portrayed the production of whole thoughts.
In the aforementioned system of the Baker '916 patent, the spatial configuration of the symbols on a given keyboard remained constant. Sequences of symbols in fixed places were consistent, allowing messages to be reliably produced with the same sequence each time. This constant mapping supported the learning of motor patterns associated with symbol sequences. As such sequences were learned, the user could establish motor programs that allow sequences to be produced quickly and accurately in the same way a touch typist efficiently spelled many words or a musician played an instrument.
The aforementioned Baker '916 patent provided an excellent means of accessing high frequency “core” vocabulary words using sequenced polysemous symbols. However, the system of the Baker '916 patent only provided limited access to the relatively large set of low frequency “fringe” vocabulary words that would only be used periodically.
A subsequent design that provided for a way to easily access fringe vocabulary utilizing non-polysemous symbols on dynamic graphical screens was disclosed in U.S. Pat. No. 5,920,303 to Baker et al. (the Baker '303 patent) issued Jul. 6, 1999, the entire contents of which are hereby incorporated herein by reference. In the system of the aforementioned Baker '303 patent, non-polysemous symbols were used for accessing fringe vocabulary. In the system of the Baker '303 patent, less than all of a plurality of keys on the displayed keyboard were dynamically redefined in response to selection of a symbol or sequentially selected symbols. These dynamic characteristics produced a dynamically redefined keyboard for accessing fringe vocabulary.
Even though the system of the aforementioned Baker '303 patent provided an improved method for accessing fringe vocabulary, it could still be limited in that it could not provide efficient access to a very large set of fringe vocabulary words. The fringe vocabularies of adolescents or adults may include many thousand words that are used on occasion in specific contexts. For example, most vocabulary words that are learned in academic or vocational settings are fringe vocabulary words related to specific topics. In the system of the Baker '303 patent, a subset of the plurality of keys on the displayed keyboard could contain dynamically re-defined non-polysemous symbols for selection of fringe vocabulary. In some embodiments, this subset of keys on the displayed keyboard may include one row of twelve keys, which may be used to access fifty different fringe words, for example. In this case, the operator may need to use command keys to browse through a set of non-polysemous symbols that is much larger than the number of available key locations. A control key may be used to dynamically redefine this subset of keys multiple times until the desired symbol is presented. In the system of the aforementioned Baker '303 patent, the operator may need to execute six or more keystrokes to access a desired fringe vocabulary word corresponding to one of a large number of non-polysemous symbols.
Although many people have learned to successfully use augmentative and alternative communication systems containing embodiments of the semantic compaction encoding techniques to communicate with very high levels of linguistic performance, some populations of children have had difficulty learning more advanced systems where the available language is effectively unlimited. Children with autism, in particular, may be overwhelmed by a large array of unfamiliar polysemous symbols and potential polysemous symbol sequences. Alternatively, they may perseverate on one symbol while a clinician is trying to provide structured instructional activities involving sequences associated with a different polysemous symbol.
The Baker '916 patent provided for embodiments that include a keyboard with a relatively large number of polysemous symbols suitable for individuals with relatively high cognitive and linguistic skills, and embodiments that included a keyboard with a relatively small number of polysemous symbols suitable for individuals with more significant cognitive or linguistic impairments. An individual who has difficulty learning the system may benefit from a system with a simpler keyboard, but this consequently limits the availability of stored language content.
U.S. Pat. No. 5,297,041 issued Mar. 22, 1994 (the Kushler '041 patent), the entire contents of which are hereby incorporated herein by reference, provided for a predictive input system which only allowed an operator to select a polysemous symbol that would lead to the valid production of a sequence for accessing a previously stored message after, an initial polysemous symbol has been selected (or even prior to selection). The subsequent Baker '303 patent provided for a dynamically re-defined keyboard, where individual keys were dynamically re-definable after one or more initial polysemous symbols were selected to provide additional visual information about potential words or messages that may be produced by selecting one or more additional symbols. Such systems may support access to thousands of stored words or messages through selection of unique sequences of polysemous symbols, and provide feedback in relation to available sequences. Although these systems may store enough content to support selective generation of numerous unique sentences with a relatively small number of polysemous symbols, all valid polysemous symbol sequences and corresponding stored words and messages are accessible to the user at all times. Thus, access to the stored language content was effectively unlimited.
In the system of the Baker '916 patent, the communication device included a keyboard coupled to a microprocessor associated with an electrical programmable read only memory. Each key of the keyboards included a relatively centrally disposed polysemous symbol, and all keys were permanently displayed in a static configuration. A given pre-stored message including one or more words, for example, could be retrieved from the electrical programmable read-only memory by actuating a specific sequence of keys that included polysemous symbols, where the key sequence was associated with the given pre-stored message.
The systems of several subsequent patents improved on the design disclosed in the Baker '916 patent while consistently maintaining a requirement that pre-stored messages be retrieved from an electronic memory via actuation of a corresponding symbol sequence using a single keyboard or keyboard overlay. The system developed by Baker, et al, and disclosed in U.S. Pat. No. 5,210,689 to Baker, et al, (the Baker '689 patent) issued on May 11, 1993, the entire contents of which are hereby incorporated herein by reference, provided for a continuous input system that allowed a user utilizing an icon or symbol mode to access stored morphemes, words, phrases, or sentences by selecting sequences of polysemous icon symbols and allowed a user utilizing a character, word prediction, or suffix mode to enter additional content using a text-based keyboard. In the system of the Baker '689 patent, the keyboards for the symbol mode and character, word prediction or suffix mode overlapped in space so that a user could automatically toggle between modes on a single keyboard without having to manually switch between modes.
The system disclosed in U.S. Pat. No. 5,920,303, issued to Baker, et al on Jul. 6, 1999 provided for a dynamic keyboard on a graphical user interface including a plurality of keys with associated symbols that are dynamically redefinable to provide access to higher level keyboards. In this system, keys could be dynamically redefined based on one or more initially selected symbols in a sequence such that symbols on dynamically redefined keys could include embellished symbols and/or newly displayed symbols. This system also allowed for easy access to core vocabulary words through selection of sequenced polysemous symbols while providing improved access to fringe vocabulary words via selection of the new or embellished symbols corresponding to fringe vocabulary words.
In the systems of the above-mentioned patents, and other patents subsequent to the Baker '916 patent, sequenced polysemous symbols were selected from a keyboard to retrieve associated stored messages from an electronic memory. Where features of the keyboard have been dynamically redefined based on one or more initially selected polysemous symbols in a sequence, those dynamically redefinable features were limited to changes in individual keys or symbols.
An alternative to the polysemous symbols disclosed in the Baker '916 patent and subsequent patents involves the use of solely non-polysemous (single meaning) symbols. When solely non-polysemous symbols are used, a small number of non-polysemous symbols, for example 20 symbols in a four by five array, are initially introduced. Selection of one of these non-polysemous symbols is typically used to produce a stored message, such as a noun that is easy to represent with pictures of a pre-stored sentence. At this point, solely non-polysemous symbols may be used easily, and with minimal effort.
Use of solely non-polysemous symbols becomes much more challenging, however, when the number of available stored vocabulary words and sentences exceeds the number of available locations on the display of the communication device; when this happens, the set of non-polysemous symbols and associated stored messages must be sorted and divided across a plurality of linked pages, and the operator must navigate between these pages to locate and select desired messages. Typically, many of the non-polysemous symbols on the highest level “master” page include links to additional pages of non-polysemous symbols. These additional pages may contain non-polysemous symbols that include links to even more pages, a link back to the master page, and/or some non-polysemous symbols that are associated with stored messages.
The words on these various pages are often organized into semantic categories. In these systems, selecting a non-polysemous symbol on the master page that represents a superordinate semantic category links the operator to a second page containing non-polysemous symbols associated with category members and/or subordinate categories. On the second page, selecting a non-polysemous symbol associated with a subordinate category activates a link to a third page containing additional non-polysemous symbols, and so on. Selecting a non-polysemous symbol associated with a category member generates the associated stored message.
As an example, the master page on a communication device may have non-polysemous symbols linking to the categories “people,” “food,” “school”, “fun times,” and clothing.” Selecting the non-polysemous symbols that includes a link to the “clothing” category may link to a page containing non-polysemous symbols associated with the category members “shirt,” “pants,” “underwear,” and “shoes,” as well as non-polysemous symbols that include links to the subordinate categories “winter clothing,” “summer clothing,” and “formal clothing,” and a non-polysemous symbol including a link back to the master page.
As the vocabulary increases in size, the number of pages of non-polysemous symbols on such a system must necessarily increase, which in turn increases the cognitive complexity of tasks involved in using the system. In order to produce any given word, the operator must recall the categorical organization of the system well enough to identify the superordinate and subordinate categories that must be linked in order to navigate to the page containing the desired word, and visually search for each symbol that should be selected during navigation.
For example, the operator may recall that “coat” is located under the superordinate category for clothing, and still not be sure whether “coat” is located under the subordinate category for “winter clothing” or “formal clothing.” This becomes a recurrent problem when navigating many pages. In order to store a 1,000 word vocabulary, consistent with that of a typically developing three-year-old, a minimum of 56 linked 20-location pages of non-polysemous symbols are required. When an operator must navigate such an extensive set of categorically organized pages, actual communication using the system may be very limited because the cognitive demands of page navigation are so extensive.
Recent developments in broadly available consumer technologies have included tablets, smart phones, and other portable multifunction devices with touch-screen interfaces, such as iPADs, iPhones, and Android devices. These devices have allowed programmers to develop and implement a wide range of third-party application programs, including assistive technology programs, as long as the program was compatible with the device hardware and the software platform that was integral to the device's operating system. The software platforms on portable multifunction devices were highly compatible with a wide range of third-party augmentative and alternative communication programs containing numerous pages of non-polysemous symbols, but were not necessarily as compatible with established embodiments of the Baker '916 patent or subsequent patents using polysemous symbol systems.
Previous embodiments of the Baker '916 patent and the subsequent patents have relied on a method of retrieving stored messages from an electronic memory when a corresponding symbol sequence is selected on a keyboard. This process could have involved a large and complex computer program with over a million lines of code and a database defining the interactions between the various polysemous or non-polysemous symbols, words, and speech. Modifying such a program to be elegantly and efficiently transferred between a plurality of operating systems on tablet, smart phone, and other portable multifunction devices may have been very difficult or even impossible.